Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: phdstudent Date: Nov 24, 2010 12:52pm
Forum: web Subject: Does a disallow in robots.txt actually delete archived content?

Suppose I have a web site without a robots.txt, which is archived by

A year later, it shows up in the index and I add a robots.txt disallowing's crawler. Predictably, stops showing the archived page.

If I then remove the robots.txt, will resume showing that old archived version some time in the future, or has that page been permanently deleted, and will never be seen again?

Thank you.